Empirical estimation of entropy functionals with confidence

Nonparametric estimation of functionals of density from finite number of samples is an important tool in domains such as statistics, signal processing and machine learning. While several estimators have been proposed in literature, the performance of these estimators is not known. We propose a kNN class of plug-in estimators for estimating non-linear functionals of density, such as entropy, mutual information and support set dimension. The plug-in estimators are designed to automatically incorporate boundary corrections for densities with finite support. Based on the statistical properties of kNN density estimators, we derive the bias and variance of the plug-in estimator in terms of the sample size, the dimension of the samples and the underlying probability distribution. We also establish a central limit theorem for the plug-in estimators. Based on these results, we specify the optimal choice of tuning parameters for minimum mean square error. The theory is illustrated by applications to problems such as intrinsic dimension estimation and structure discovery in high dimensional data.